Gustavo Diaz
McMaster University
gustavodiaz.org
diazg2@mcmaster.ca
Slides: talks.gustavodiaz.org/tec
Explicit: Is a little bias worth the increase in precision?
Implicit: Improving precision without sacrificing unbiasedness?
Explicit: Is a little bias worth the increase in precision?
Implicit: Improving precision without sacrificing unbiasedness?
Here is a list of things that some people have done.
Please listen to them and then tell me HOW MANY of them you have done in the past two years.
Do not tell me which ones. Just tell me HOW MANY:
Do not tell me which ones. Just tell me HOW MANY:
\[ \text{Proportion(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]
Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?
\[ \text{Proportion(Voted yes)} =\\ \text{Mean(Voted yes)} \]
List A
List B
Organization X (advocating for immigration reduction and measures against undocumented immigration)
Randomly appears in list A or B
Single list: Half of the respondents see sensitive item
Double list: Everyone sees it
Equivalent to two parallel list experiments
\[ \hat{\tau}_A = \text{Mean}(A_t) - \text{Mean}(A_c) \]
\[ \hat{\tau}_B = \text{Mean}(B_t) - \text{Mean}(B_c) \]
\[ \hat{\tau}_{Pooled} = (\hat{\tau}_A + \hat{\tau}_B)/2 \]
Baseline lists need to be comparable
Easiest way is to use paired items
American Family Association (A) \(\approx\) Tea Party Patriots (B)
BUT that makes it easier to spot the sensitive item
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
| List order | Sensitive item location |
|---|---|
| Fixed | Fixed |
| Randomized | Fixed |
| Fixed | Randomized |
| Randomized | Randomized |
Design effect(Blair and Imai 2012)
The inclusion of a sensitive item affects how survey participants respond to the baseline items within the list.
Carryover design
The inclusion of a sensitive item in one list affects how participants respond to the baseline items in the other list.
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
| Observed response | List 1 | List 2 | Difference |
|---|---|---|---|
| Baseline | 2 | 2 | 0 |
| Deflation | |||
| Sensitive first | 1 | 1 | 0 |
| Sensitive second | 2 | 1 | 1 |
| Inflation | |||
| Sensitive first | 3 | 3 | 0 |
| Sensitive second | 2 | 3 | -1 |
Goal: Detect asymmetric shift across treatment schedules
Two tests:
Difference-in-differences (stacked responses)
Signed-rank test (paired responses)
Goal: Detect asymmetric shift across treatment schedules
Two tests:
Difference-in-differences (stacked responses)
Signed-rank test (paired responses)